Appl. No: 09/742,165 

Amdt. Dated May 4, 2006 

Reply to Office Action of Nov. 4 t 2005 

REMARKS/ARGUMENTS 

Prior to this Amendment, claims 1-26 were pending in the application. 

Claim 1 is amended to include the limitations of claim 2 to stress that the 
media streams include streaming video that is mixed, and other language is added 
to stress that synchronization is based on time including the variable transmission 
delay from the first and second sources to the receiving interface (e.g., rather than 
involving creating a variable transmission rate at a media source as shown in at 
least one of the cited references). Claim 2 is cancelled and claim 3 is amended to 
correct dependency. 

Independent claim 15 is amended similar to claim 1 to stress that the claimed 
apparatus is useful for mixing video streams based on transmission delays over a 
varying network. The limitations of claim 16 are inserted into claim 15, and claim 16 
is cancelled. Claim 17 is amended to correct dependency. 

Independent claim 20 is amended to include the limitations of dependent 
claims 21 and 24, which are cancelled, and, as amended, the claimed method calls 
for mixing media streams with video files into a "synchronized media stream" 
"wherein the first and second media streams are presented in the synchronized 
media stream concurrently." Such mixing of two media streams including video for 
concurrent presentation is not shown or suggested by the cited references. 

Claims 27-29, which depend from claim 20, are added to further highlight the 
mixing of two video streams based on time to achieve traditional television effects 
with video streamed over subnetworks that make up the Internet or similar 
communications networks (with support found at least on page 10 of Applicant's 
specification). New claim 30 is added to protect the concept of time-based 
synchronization of two live webcasts as discussed at least in lines 15-21 of page 21 
and not shown in the cited references. 

After entry of the Amendment, claims 1, 3-15, 17-20, 22, 23, and 25-30 
remain for consideration by the Examiner. 
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Rejections under 35 U.S.C. 102 

In the November 4, 2005 Office Action, claims 1-4, 8, 1 1-21 , and 23-26 were 

rejected under 35 U.S.C. 102(b) as being anticipated by U.S. Pat. No. 5,844,600 
("Kerr"). This rejection is traversed based on the following remarks. 

As amended, claim 1 calls for an input interface that receives first and 
second media streams that each include a streaming video portion. A controller 
determines a "variable transmission delay for the first and second media streams 
from the first and second media sources to the input interface and performs the 
selective retrieving based on the determined variable transmission delays." 
Additionally, the controller mixes the retrieved time-adjusted streams "into a 
composite media stream wherein the first and second time-adjusted streams are 
synchronized based on time." Kerr fails to teach or suggest each of these 
limitations of claim 1 . 

Before turning to Kerr, it may be useful to discuss a few unique benefits of 
the claimed limitations. The system of claim 1 is effective for mixing streams with 
video portions that are streamed over differing networks and/or different network 
paths. Audio synchronization to video is much less demanding as the human ear 
typically can only detect errors that are in the range of 20 to 40 milliseconds or 
higher. In contrast, video time-based synchronization requires synchronization in 
the 100 nanosecond or lower range with higher definition requiring even higher 
levels of synchronization. In part, this is because video mixing may involve 
superimposition of two images or fading one into another or cutting a hole and then 
matting another into the hole (e.g., mixing, matting, keying, and the like). For 
example, a live stream of a news or weather anchor may be keyed into a hole 
created in another live stream (e.g., a stream of live action video or weather map or 
the like) during mixing operations. Such mixing may involve mapping video streams 
on a pixel by pixel basis or even fractions of pixels (e.g., image element in one 
stream to an image element in another stream). Such video mixing is often orders 
of magnitude more accurate than that required to match an audio signal to a video 
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signal. Prior to the invention, the problems associated with such mixing of media 
streams transmitted over the Internet and other networks had not been addressed, 
and one solution was to simply display video streams separately (e.g., avoid mixing) 
or at most try to synchronize a sound stream with a video stream but again without 
mixing. 

Kerr provides the latter type of solution by teaching a technique for time- 
based synchronization of a video stream with an audio stream - that are maintained 
as separate streams. Specifically, Kerr fails to teach receiving two media streams 
from two sources that each include streaming video portions, determining a variable 
transmission delay for each stream, retrieving the two streams from a buffer based 
on the determined delays, and then mixing the two streams into a composite media 
stream. 

As to the claim element of receiving two media streams that both include 
streaming video portions, the Office Action states that the streams shown by Kerr 
are all video or all audio. Applicant agrees that Kerr fails to show receiving two 
media streams that include streaming video portions and later mixing the two 
streams (with the lack of the mixing element discussed more below). Kerr in fact 
teaches receiving audio information and corresponding video information (not video 
information and video information in two streams). For this reason along, Kerr fails 
to anticipate the system of claim 1 . 

Kerr also fails to show retrieving two streams including video portions from a 
buffer based on variable transmission delays determined for the two streams. The 
Office Action cites Kerr at col. 10, lines 1-20 for determining such a delay and col. 9, 
lines 15-22 and lines 55-65 for teaching these features of a controller. But, as 
discussed by the Examiner, the Kerr controller discusses delays to either a video 
stream or an audio stream and synchronization based on such delays. There is no 
discussion of retrieving two streams from buffers that each include a video portion 
based on delays in the media streams during transmission over a network. For this 
additional reason, claims 1 is not anticipated by Kerr. 
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Likely the most easily seen difference between the teaching of Kerr and the 
system of claim 1 is that Kerr fails to teach mixing two streams containing video 
portions into "a composite media stream wherein the first and second time-adjusted 
streams are synchronized based on time. In fact, Kerr teaches "switching" between 
video streams from varying sources and synchronizing an audio with the displayed 
video stream. This is discussed in the Summary at col. 2, lines 48-62 in which "the 
separate audio and video signals are processed and synchronized by the interface 
module of the destination and provided to the audio/video terminal." In other words, 
the audio signal is kept separate from the video signal but the two separate signals 
are synchronized (e.g., see the third example discussed in col. 3, lines 21-31 of 
using time stamps in the audio and the video signals). At col. 4, lines 57-65, Kerr 
discusses again that the two signals are received separately and the video 
information is sent to video monitor and the audio to a speaker using one of the 
synchronization techniques. 

There is no discussion of mixing the synchronized audio and video 
information let alone mixing two media streams with video portions as called for in 
the last element of claim 1. This point is stressed in Kerr at col. 7, lines 64 and on 
with the statement that because "the amount of video data is large, the invention 
advantageously requires that the video and audio data streams be separately 
handled (i.e., that they not be multiplexed)." Hence, the mixing element of claim 1 
is not only not taught or suggested but, instead, Kerr actually teaches away from the 
mixing of claim 1 (e.g., Kerr provides no motivation to modify its teaching to include 
mixing of an audio and video stream let alone mixing of two video streams in "the 
amount of video data is large"). The Office Action cites col. 10, lines 20-23 for 
teaching such mixing, but Kerr is only discussing passing the synched audio and 
video information to the audio and video systems (and as discussed throughout 
Kerr the information is handled and maintained separately). For this additional 
reason, Kerr fails to anticipate the system of claim 1 . 
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Claims 3, 4, 8, and 11-14 depend from claim 1 and are believed allowable 
over Kerr at least for the reasons provided for allowing claim 1 . Claims 3 and 4 are 
addressed to the two streams having differing compression formats as received and 
handling the streams to allow mixing by decoding into compatible forms. The Office 
Action cites Kerr at col. 10, lines 40-41, col. 4. lines 45-50, col. 5, lines 55-60, and 
col. 6, lines 5-17 and lines 60-67. However, Kerr teaches an audio stream and a 
video stream that are handled separately. There is no discussion of decoding the 
two streams into formats that are compatible prior to storage in buffers. Applicant 
asks Examiner to point to a device in Kerr that decodes the audio and video 
streams or to withdraw the rejection (with col. 10, lines 40 and 41 simply mentioning 
some useful compression standards but does not discuss putting the video and 
audio stream into one of these so they can be mixed). For this additional reason, 
claims 3 and 4 are believed allowable over Kerr. 

Claim 1 1 calls for the composite stream to be mixed in a manner that the two 
video portions create picture-in-picture or side by side portions. As discussed 
above, Kerr fails to teach mixing of two video streams and therefore, cannot teach 
providing the specific mixing called for in claim 1 1 . The Office Action cites Kerr at 
col. 12, lines 40-45 for teaching this limitation. At this citation, Kerr states "the video 
cross-connect switch 152 routes the video data, and does not forward it to the 
processing section 401 of the bridge. However, if video processing such as quad- 
split or continuous presence is required, the switched video data is forwarded 
through the video ports 453a, 453b, ...to the video processing unit 154 which is 
directly under the control of the session management and control block..." 
Applicant disagrees that this teaches providing picture-in-picture or side by side 
mixing of two video streams that are first synchronized based on time delays over a 
network. The video data is described as "switched" and no mixing of time 
synchronized streams is taught as being performed by the video processing unit 
154. For this additional reason, claim 11 is believed allowable over Kerr. 
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Claim 14 calls for an end-user node to include a synchronizer for performing 
time-based correction of the previously synchronized and mixed "composite media 
stream" to adjust for time delay between the device that does the mixing in a time- 
based fashion and the end-user (e.g., delays in the milliseconds can occur even 
within a LAN or the like). Kerr does not show this additional feature of the claimed 
system. The Office Action cites col. 10, lines 1-25 of Kerr, but Applicant could find 
no teaching in this portion of Kerr that a time correction was done after the original 
time synchronization between the separate audio and video streams by the Kerr 
controller. Hence, Applicant requests that the Examiner indicate which parts of the 
Kerr system are providing the claimed function or withdraw the rejection. 

Independent claim 15 includes limitations similar to claim 1 including 
receiving and mixing two media streams made up of video stream portions. Hence, 
the reasons provided for allowing claim 1 over Kerr are believed applicable to claim 
15. Additionally, claim 15 includes limitations similar to dependent claims 3 and 4, 
and the reasons for allowing claims 3 and 4 over Kerr are applicable to claim 15. 
Specifically, the Examiner is requested to show in Kerr where a decoder decodes 
two video streams into compatible formats (which is useful for facilitating mixing 
which as discussed with reference to claim 1 is not performed by Kerr). 

Claims 17-19 depend from claim 15 and are believed allowable over Kerr at 
least for the reasons provided for claim 15. Claim 18 calls for concurrent delivery of 
time-adjusted first and second streams in a composite media stream, and as 
discussed above for video streams this typically involves a resolution or even pixel- 
by-pixel mapping or the like not shown by Kerr as this reference fails to teach 
mixing of audio and video streams (let alone video and video). Claim 19 calls for 
the time of receipt to be determined based on an external timing reference signal. 
Kerr is cited at col. 10, lines 1-20 for this feature of the claimed apparatus but, at 
this citation, Kerr teaches determining delays but does not teach using an external 
timing reference and the figures do not show an external device transmitting such 
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signals. For these additional reasons, claims 18 and 19 are believed allowable over 
Kerr. 

Independent claim 20 was rejected for "the same reasons as rejection to 
claim 1 above" in the Office Action. As amended claim 20 calls for the two media 
streams to include data packets from one or more video files and adjusting the two 
streams including "matching the data packets of the first and second media streams 
based on transmittal times from the first and second media sources." Further, a 
"synchronized media stream" is created "by mixing the first and the second media 
streams" such that the streams are presented "concurrently." As discussed with 
reference to claim 1 , Kerr fails to show any mixing of video streams, and as a result, 
the reference cannot anticipate claim 20. Further, claim 20 requires matching of 
data packets of the two streams and this is not shown by Kerr for two video streams 
(or even for the synchronized (but separate) video and audio streams). This is not 
shown, as suggested in the rejection of claim 24, by Kerr at col. 10, lines 1-27. 
Hence, claim 20 is not anticipated (or even suggested) by Kerr, and Applicant 
requests that the rejection be withdrawn. 

Claims 23, 25, and 26 depend from claim 20 and are believed allowable over 
Kerr for at least the reasons provided for allowing claim 20. Claims 25 and 26 have 
limitations similar to claims 3 and 4 and are believed allowable for the reasons 
provided for these claims. 

Additionally, in the Office Action, claims 1-10, 12-13, 15, and 20-26 were 
rejected under 35 U.S.C. 102(b) as being anticipated by U.S. Pat. No. 4,598,397 
("Nelson"). This rejection is respectfully traversed based on the amendments to the 
claims and the following remarks. 

With regard to claim 1 , Nelson fails to teach determining "variable 
transmission delay" "from the first and second media sources to the input interface" 
that receives the first and second media streams. Nelson in contrast is directed to 
selectively creating delays at a source so as to account for variable transmission 
"rates" over a communication line. Nelson's concept is as discussed in col. 4, lines 
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13-32 to provide a technique for supporting "devices operating at different data 
rates" and this is achieved as discussed at col. 4, lines 53-61 "synchronize the data 
rates of that information with the network data rate." In other words, the technique 
involves trying to control the transmission of data at rates that better match the 
transmission rate over a network section. This interpretation of the teaching of 
Nelson is backed by a study of col. 17, line 20 to col. 20, line 41 (see, for example, 
col. 19, lines 18-21 which discusses monitoring and controlling "the output rates"). 

Hence, Nelson fails to teach an interface on a network that receives two 
video streams, determines a delay caused by its transmission over a network, and 
then acts to mix the two video streams to account for the varying travel delay. 
Controlling of transmission of streams of data at a source can be useful to 
overcome gross or large transmission delays or variances in transmission rates but 
is not effective at providing the high level of resolution typically needed for mixing 
video streams (e.g., pixel-to-pixel or image element-to-image element mapping). To 
this end, the system of claim 1 addresses differing travel times over a network for 
video streams at the receiving device rather than just at the source. Since each and 
every feature of the claim are not shown (for example, the "delays" discussed in 
Nelson are being created/controlled by the Nelson devices to suit transmission data 
rates not based data received over a network from two sources), claim 1 is not 
anticipated by Nelson. 

Claims 3-10, 12, and 13 depend from claim 1 and are believed allowable 
over Nelson at least for the reasons provided for allowing claim 1 over this 
reference. Additionally, claims 3 and 4 as discussed with reference to Kerr call for 
the two streams to be received in differing compression formats and then be altered 
for mixing. Nelson is cited at col. 5 in its object section where it discusses the 
various data and its form that may be transmitted to local devices but there is no 
discussion of receiving two video streams in differing compression formats, 
decoding/changing to a neutral format for buffering, and then mixing. 
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Claim 5 discusses mixing such that the two streams are mixed for sequential 
delivery without "dead air" and this is not shown by Nelson at col. 9, lines 50-52 
which simply states that the "format of the data character received from the system 
node is typically a start bit, a six-bit character and three stop bits, for a total often 
bits per character." This fails to even suggest the type of mixing called for in claim 
5. 

Independent claim 1 5 includes limitations similar to that of claim 1 and the 
reasons provided for allowing claim 1 over Nelson are believed applicable to claim 
15. Further, claim 15 includes limitations similar to dependent claims 3 and 4, and 
the reasons for allowing claims 3 and 4 over Nelson are applicable to claim 15. 
Specifically, the Examiner is requested to show in Nelson where a decoder decodes 
two video streams into compatible formats. 

Independent claim 20 was rejected for "the same reasons as rejection to 
claim 1 above." As amended claim 20 calls for the two media streams to include 
data packets from one or more video files and adjusting the two streams including 
"matching the data packets of the first and second media streams based on 
transmittal times from the first and second media sources." Further, a 
"synchronized media stream" is created "by mixing the first and the second media 
streams" such that the streams are presented "concurrently." As discussed with 
reference to claim 1 , Nelson fails to show any mixing of video streams based on 
determined "variable transmission delays" between two sources and the receiving 
interface, and as a result, the reference cannot anticipate claim 20. Further, claim 
20 requires matching of data packets of the two streams and this is not shown by 
Nelson for two video streams. Hence, claim 20 is not anticipated (or even 
suggested) by Nelson, and Applicant requests that the rejection be withdrawn. 

Claims 23, 25, and 26 depend from claim 20 and are believed allowable over 
Nelson for at least the reasons provided for allowing claim 20. Additionally, claims 
25 and 26 have limitations similar to claims 3 and 4 and are believed allowable for 
the reasons provided for these claims. 
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Rejections under 35 U.S.C. 103 

Further, in the November 4, 2005 Office Action, claims 5 and 6 were rejected 

under 35 U.S.C. 103(a) as being unpatentable over Kerr. This rejection is traversed 
based on the following remarks. 

Claims 5 and 6 depend from claim 1 and are believed allowable as 
depending from an allowable base claim. Claim 5 calls for the composite media 
stream to be formed sequentially with one stream provided directly after the other 
with no "dead air." Kerr is teaching synching of audio with a video stream for 
teleconferencing. There is no teaching of providing a composite stream where the 
streams are mixed beginning to end and so on. Hence, Kerr fails to teach claim 5. 
Claim 6 calls for the use of an external timing reference and, as discussed with 
reference to claim 19, Kerr fails to show this in its figures or refer to such a 
technique in the cited portions of its text. For these additional reasons, claims 5 and 
6 are believed allowable over Kerr. 

Also, in the Office Action, claims 7-9 and 22 were rejected under 35 U.S.C. 
103(a) as being unpatentable over Kerr in view of U.S. Pat. No. 6,934,759 ("Hejna"). 
This rejection is traversed based on the following remarks. 

Claims 7-9 depend from claim 1 and claim 22 depends from claim 20. 
Claims 1 and 20 are believed allowable over Kerr for the reasons provided above. 
Hejna does not over the deficiencies of Kerr with respect to claims 1 and 20, and as 
a result, claims 7-9 and 22 are believed allowable over the combined teaching of 
Kerr and Hejna. 

Further, claim 11 was rejected under 35 U.S.C. 103(a) as being unpatentable 
over Nelson in view of "Applicant Admitted Prior Art (AAPA)" yet further in view of 
"First Generation of True HDTV Component Products and Advanced Line of Analog 
Projection Televisions Unveiled by Mitsubishi Computer Electronics America" 
("Mitsubishi"). 

Claim 1 1 depends from claim 1 which is believed allowable over Nelson for 
the reasons provided above and the AAPA and Mitsubishi reference fail to over 
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come the deficiencies of Nelson with reference to claim 1 . It should be understood 
that Applicant is not claiming the invention of PIP or split screen technologies by 
themselves but instead is claiming the mixing of video streams over the Internet and 
similar networks using techniques that allow synchronization to an accuracy and 
resolution (e.g., pixel-by-pixel mapping) that allows two streams to be combined to 
provide these effects. The AAPA does not discuss PIP or side by side in this 
manner nor does the Mitsubishi reference. Applicant requests that the Examiner 
provide a reference that shows two video streams being transmitted over the 
Internet or other similar digital communications network, time synchronized as 
called for in the claims, and then mixed to create these effects or to withdraw this 
rejection as unsupported (e.g., rather than simply keeping the video separate or 
accepting the unsynchronized characteristics of streaming data on such networks). 

Still further, claims 16-19 were rejected under 35 U.S.C. 103(a) as being 
unpatentable over Nelson in view of U.S. Pat. No. 6,1 15,422 ("Anderson"). This 
rejection is traversed based on the following remarks (with claim 16 being 
cancelled). Claims 17-19 depend from claim 15 are believed allowable as 
depending from an allowable base claim. Specifically, claim 15 is believed 
allowable over Nelson for the reasons provide above, and Anderson fails to 
overcome the deficiencies of Nelson discussed with reference to claim 15. 

Conclusions 

In view of all of the above, it is requested that a timely Notice of Allowance be 
issued in this case. 

A check is provided for the fee associated with a request for a 3-month 
extension of time. Any fee deficiency associated with this submittal may be charged 
to Deposit Account No. 50-1123. 


May 4. 2006 



Kent A. Lembke, Reg. No. 44,866 
(720) 406-5378 Tel 


19 


\\\BO - 82705/0001 - 187167 v1 


